poc: Extensible Mapper using JavaScript #3650

didier-wenzek · 2025-05-28T11:21:36Z

Proposed changes

This POC explores quickJS as an alternative to:

WASM gen mapper POC: tedge wasm mapper #3517
RustyScript gen mapper poc: Generic Mapper using JavaScript #3626

A thin-edge mapper that can be extended by users with mapping rules implemented in JavaScript.

Features:

Chain stateless filters along pipelines: MQTT sub| filter-1 | filter-2 | ... | filter-n | MQTT pub
Configure filters
Update filter config using MQTT
Implement state-full filters (deferring messages up to the end of a time window)
Combine user-provided JS filters with built-in Rust filters

Criteria:

Expressive power?
Dev tooling ?
Performance? Memory footprint?
Debuggability?

Examples:

Add missing timestamps
Discard measurements with timestamp out of the expected range (too old or in the future)
Translate thin-edge json measurment into cumulocity json
Add measurement units - read from topic metadata
Translate Collectd data into thin-edge JSON
Group messages received during the same time window
Compute 5-minute rolling average, min, max
A circuit breaker filter detecting that too many messages are being sent within a given period
See also real use-case examples

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
Documentation Update (if none of the other choices apply)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

I have read the CONTRIBUTING doc
I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
I ran just format as mentioned in CODING_GUIDELINES
I used just check as mentioned in CODING_GUIDELINES
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

Further comments

codecov · 2025-05-28T11:31:24Z

Codecov Report

Attention: Patch coverage is 79.43925% with 22 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/extensions/tedge_mqtt_ext/src/lib.rs	80.23%	13 Missing and 4 partials ⚠️
crates/common/mqtt_channel/src/connection.rs	50.00%	2 Missing ⚠️
crates/core/tedge_mapper/src/lib.rs	0.00%	2 Missing ⚠️
crates/core/plugin_sm/src/operation_logs.rs	0.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

didier-wenzek · 2025-05-28T14:23:58Z

The javascript examples have to be updated.

For instance the syntax config.topic || "te/device/main///m/collectd" is not supported. See fbe1341

reubenmiller · 2025-05-28T17:08:14Z

The javascript examples have to be updated.

For instance the syntax config.topic || "te/device/main///m/collectd" is not supported. See fbe1341

Though it seems that Optional Chaining is part of ES2020 (which means it should be supported by QuickJS, I confirmed on the WASM version of QuickJS at least), so that would be the most similar to your previous syntax (just adding a ? before the .).

For example:

let topic = config?.topic || "te/device/main///m/collectd";

albinsuresh

Here is my first round of review comments focussing purely on the pipeline APIs and not much of the rust infra code. Unfortunately, I'm having some trouble testing the feature due to some certificate misconfiguration preventing the mapper from even starting with the following error:

Jun 10 06:36:23 2a9636a9acc5 tedge-mapper[3107]: 2025-06-10T06:36:23.630655575Z ERROR Runtime: Actor MQTT-2 has finished unsuccessfully: ActorError(InvalidPrivateKey(InvalidCertificate(Other(OtherError(UnsupportedCertVersion)))))

Will provide additional usability feedback once that issue is resolved.

albinsuresh · 2025-06-09T17:48:06Z

docs/src/references/mappers/gen-mapper.md

+stages = [
+    { filter = "add_timestamp.js" },
+    { filter = "drop_stragglers.js", config = { max_delay = 60 } },
+    { filter = "te_to_c8y.js", meta_topics = ["te/+/+/+/+/m/+/meta"] }


Wondering if stage-wise meta_topics is really necessary or a pipe-line level meta_topics would be sufficient (and simpler to maintain).

What we want here is for the messages received on the meta topics to be directly delivered to the stage. The other stages have no way to do something meaningful with these messages (In this case, units for various measurement types).

A better name might be config_topics.

albinsuresh · 2025-06-09T18:40:13Z

docs/src/references/mappers/gen-mapper.md

+stages = [
+    { filter = "add_timestamp.js" },
+    { filter = "drop_stragglers.js", config = { max_delay = 60 } },
+    { filter = "te_to_c8y.js", meta_topics = ["te/+/+/+/+/m/+/meta"] }


Expressing the correlation between the input_topic and meta_topic with wildcards appears slightly inaccurate/misleading. For example, when the above definition is applied on a te/+/+/+/+/m/temperature measurement, the corresponding meta_topic should have only been te/+/+/+/+/m/temperature/meta. But with the wildcard te/+/+/+/+/m/+/meta subscription, it covers the meta topics of other measurements like m/pressure/meta, m/humidity/meta etc as well. Wondering if we should introduce some sort of variable substitution syntax like m/{m_type} and m/{m_type}/meta so that this correlation is better expressed.

What you propose seems too specific to this specific example (adding units to measurements leveraging the meta topics as defined by thin-edge). Sure the script (here te_to_c8y.js) has to appropriately match data and meta data, but why should the generic mapper enforce that from the outside?

docs/src/references/mappers/gen-mapper.md

albinsuresh · 2025-06-10T06:55:07Z

Unfortunately, I'm having some trouble testing the feature due to some certificate misconfiguration preventing the mapper from even starting with the following error:
Jun 10 06:36:23 2a9636a9acc5 tedge-mapper[3107]: 2025-06-10T06:36:23.630655575Z ERROR Runtime: Actor MQTT-2 has finished unsuccessfully: ActorError(InvalidPrivateKey(InvalidCertificate(Other(OtherError(UnsupportedCertVersion)))))
Will provide additional usability feedback once that issue is resolved.

Though I couldn't fix the cert issue, I've moved on by completely disabling auth settings.

didier-wenzek · 2025-06-20T08:50:41Z

crates/extensions/tedge_gen_mapper/pipelines/loop.toml

+
+stages = [
+    { filter = "add_timestamp.js" },
+    { filter = "circuit-breaker.js", tick_every_seconds = 1, config = { stats_topic = "te/error", too_many = 10000, message_on_too_many = { topic = "te/device/main///a/too-many-messages", payload = "too many messages" }, message_on_back_to_normal = { topic = "te/device/main///a/too-many-messages", payload = "back to normal" } } }


This line highlights pain points with the current configuration format.

There are two levels of config which is confusing. tick_every_second is used by the engine, while config.too_many is passed to the script. Flattening the config might be simpler.

TOML inline tables are intended to appear on a single line. Leading to painfully long lines as here.

Passing a JSON payload to the filter config is not straightforward. Currently the engine expects a string and the JS filter has to stringyfy the payload.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

Signed-off-by: reubenmiller <reuben.d.miller@gmail.com>

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

For now, the bindgen feature is enable for all targets. One could consider to use bindings shipped with quickjs when available. See https://github.com/delskayn/rquickjs?tab=readme-ov-file#supported-platforms Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

…le quickjs

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

didier-wenzek requested review from albinsuresh, jarhodes314, rina23q and reubenmiller as code owners May 28, 2025 11:21

didier-wenzek had a problem deploying to Test Pull Request May 28, 2025 11:21 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request May 28, 2025 14:19 — with GitHub Actions Failure

didier-wenzek marked this pull request as draft May 28, 2025 14:21

jarhodes314 had a problem deploying to Test Pull Request May 30, 2025 15:27 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request June 3, 2025 08:52 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request June 3, 2025 08:57 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request June 3, 2025 09:01 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request June 3, 2025 13:45 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request June 4, 2025 17:20 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request June 4, 2025 17:21 — with GitHub Actions Failure

This was referenced Jun 5, 2025

poc: Generic Mapper using JavaScript #3626

Closed

POC: tedge wasm mapper #3517

Closed

didier-wenzek temporarily deployed to Test Pull Request June 6, 2025 08:50 — with GitHub Actions Inactive

didier-wenzek had a problem deploying to Test Pull Request June 6, 2025 17:42 — with GitHub Actions Failure

albinsuresh reviewed Jun 10, 2025

View reviewed changes

didier-wenzek had a problem deploying to Test Pull Request June 19, 2025 14:19 — with GitHub Actions Failure

didier-wenzek had a problem deploying to Test Pull Request June 20, 2025 08:41 — with GitHub Actions Failure

didier-wenzek commented Jun 20, 2025

View reviewed changes

didier-wenzek had a problem deploying to Test Pull Request June 20, 2025 14:40 — with GitHub Actions Failure

didier-wenzek force-pushed the poc/tedge-quickjs-mapper branch from fba7bdd to 2e41d5e Compare June 24, 2025 18:38

didier-wenzek requested a deployment to Test Pull Request June 24, 2025 18:38 — with GitHub Actions Waiting

didier-wenzek force-pushed the poc/tedge-quickjs-mapper branch from 2e41d5e to e759966 Compare June 24, 2025 18:45

didier-wenzek and others added 10 commits July 22, 2025 21:45

fixup! fixup! Flow API: Use more idiomatic Javascript names

4a8a44f

Naming fix: tedge-mapper gen -> tedge-mapper flows

86a9c67

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

Naming fix: /etc/tedge/gen-mapper -> /etc/tedge/flows

062df85

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

fixup! Naming fix: /etc/tedge/gen-mapper -> /etc/tedge/flows

627caa4

fixup! Naming fix: tedge-mapper gen -> tedge-mapper flows

dd36cac

add unit test for async onMessage function

0efeadc

Signed-off-by: reubenmiller <reuben.d.miller@gmail.com>

fixup! add unit test for async onMessage function

5ba10ff

Improve error reporting on JS loading errors

dc2696c

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

Set timeout limits to flow scripts

8212eac

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

fixup! Improve error reporting on JS loading errors

f2ca7b5

didier-wenzek force-pushed the poc/tedge-quickjs-mapper branch from 5d6a90a to f2ca7b5 Compare July 22, 2025 19:55

didier-wenzek temporarily deployed to Test Pull Request July 22, 2025 19:55 — with GitHub Actions Inactive

didier-wenzek temporarily deployed to Test Pull Request July 23, 2025 07:39 — with GitHub Actions Inactive

fixup! Use bindgen to compile quickjs

bfcb49b

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 13:01 — with GitHub Actions Waiting

fixup! fixup! Use bindgen to compile quickjs

7fd29f4

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 13:16 — with GitHub Actions Waiting

fixup! fixup! fixup! Use bindgen to compile quickjs

669f5ed

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 15:36 — with GitHub Actions Waiting

fixup! fixup! fixup! fixup! Use bindgen to compile quickjs

9f08006

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 15:51 — with GitHub Actions Waiting

fixup! fixup! fixup! fixup! fixup! Use bindgen to compile quickjs

03a966f

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 16:00 — with GitHub Actions Waiting

fixup! fixup! fixup! fixup! fixup! fixup! Use bindgen to compile quickjs

8cc0fd8

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 16:15 — with GitHub Actions Waiting

fixup! fixup! fixup! fixup! fixup! fixup! fixup! Use bindgen to compi…

939e0ca

…le quickjs

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 16:29 — with GitHub Actions Waiting

Nth fixup! Use bindgen to compile quickjs

883feaa

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>

didier-wenzek requested a deployment to Test Pull Request July 23, 2025 16:47 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

poc: Extensible Mapper using JavaScript #3650

poc: Extensible Mapper using JavaScript #3650

Uh oh!

didier-wenzek commented May 28, 2025 •

edited

Loading

Uh oh!

codecov bot commented May 28, 2025

Uh oh!

didier-wenzek commented May 28, 2025

Uh oh!

reubenmiller commented May 28, 2025

Uh oh!

albinsuresh left a comment

Uh oh!

albinsuresh Jun 9, 2025 •

edited

Loading

Uh oh!

didier-wenzek Jun 11, 2025

Uh oh!

albinsuresh Jun 9, 2025

Uh oh!

didier-wenzek Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albinsuresh commented Jun 10, 2025

Uh oh!

didier-wenzek Jun 20, 2025

Uh oh!

Uh oh!

poc: Extensible Mapper using JavaScript #3650

Are you sure you want to change the base?

poc: Extensible Mapper using JavaScript #3650

Uh oh!

Conversation

didier-wenzek commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Types of changes

Paste Link to the issue

Checklist

Further comments

Uh oh!

codecov bot commented May 28, 2025

Codecov Report

Uh oh!

didier-wenzek commented May 28, 2025

Uh oh!

reubenmiller commented May 28, 2025

Uh oh!

albinsuresh left a comment

Choose a reason for hiding this comment

Uh oh!

albinsuresh Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

didier-wenzek Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

albinsuresh Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

didier-wenzek Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albinsuresh commented Jun 10, 2025

Uh oh!

didier-wenzek Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

didier-wenzek commented May 28, 2025 •

edited

Loading

albinsuresh Jun 9, 2025 •

edited

Loading